The dedicated x86-pytorch-build-runner fleet is defined in
DLContainersInfraCDK mainline (commit 85a06e065, rebalanced by
f4bcbc9c on 2026-06-30) - identical hardware to x86-vllm-build-runner
and x86-sglang-build-runner, but reserved for pytorch.

Route all pytorch builds (both CPU and GPU) to the dedicated fleet
unconditionally, matching the vllm.pipeline.yml and sglang.pipeline.yml
pattern. This replaces the temporary conditional that routed GPU builds
to x86-vllm-build-runner as a stopgap.

Cache locality on a dedicated fleet is expected to fix the wheel-export
OOM: BuildKit finds the primary build's flash-attn + TE layers still
cached on the same host, so the wheel-export re-run reduces to a
COPY --from=<cached-layer> instead of a concurrent recompile.

- pytorch.pipeline.yml: unconditional fleet:x86-pytorch-build-runner
- ci-architecture.md: document the new fleet row

PR-6365 ECR vulnerability scan flagged 2 HIGH CVEs on 2.13 CPU
images (also on SageMaker CPU variant). Both are unbounded-memory /
catastrophic-backtracking bugs in soupsieve's CSS selector parser,
fixed upstream in 2.8.4.

soupsieve is transitive via beautifulsoup4 -> python-fasthtml ->
fastai. Add an explicit direct pin `soupsieve>=2.8.4` to both cpu
and cuda pyproject.toml (following prior transitive-CVE bump
convention: msgpack, tornado, cryptography, aiohttp), regenerate
uv.lock via `uv lock`.

CVEs cleared:
- GHSA-2wc2-fm75-p42x (unbounded memory on large selector lists)
- GHSA-836r-79rf-4m37 (regex catastrophic backtracking on attribute selectors)

Upstream fix pin: soupsieve>=2.8.4

The wheel-cache upload path re-runs `docker buildx build --target
wheel-export` to extract compiled wheels from the wheel-export stage.
Because that re-invocation passed NO --build-arg while the primary
build passes ~15 of them, BuildKit computed a different cache key for
every stage and had to recompile flash-attn + TE from source
concurrently, OOM-killing the CodeBuild runner (exit 137 at
upload_wheels.sh:45).

Forward the same --build-args the primary build uses (read from
EXTRA_BUILD_ARGS env var, the space-separated list of arg names the
build-image action already computes via resolve_build_args.py). The
wheel-export stage now hits BuildKit's local cache for every prior
stage and reduces to a COPY --from=<cached-layer>, avoiding the
recompile entirely.

Reverts commit 2705344. That change forwarded primary-build --build-args
to the wheel-export re-invocation to make BuildKit cache-hit and avoid
recompiling flash-attn + TE.

No longer needed: the x86-pytorch-build-runner fleet has been bumped from
m6a.8xlarge (128 GB) to m6a.12xlarge (192 GB) via DLContainersInfraCDK CR
CR-288126860. The runner now has enough memory headroom to complete the
cold-cache wheel-export recompile without OOM.

Reverting keeps upload_wheels.sh in its upstream shape for other
frameworks that share the pattern (though currently only pytorch_runtime
uses this exact copy).

Starting with EFA 1.48, the installer auto-detects NGC containers via
/opt/nvidia/nvidia_entrypoint.sh (present in nvidia/cuda:*-amzn2023 base
images) and skips installing the AL2023 libnccl-ofi RPM. This left the
pytorch runtime image without libnccl-net.so, breaking every unit /
sanity / single-gpu / efa test that asserts the OFI plugin is present.

The sibling scripts/docker/common/install_efa_amzn2023.sh already has
this fix (lines 37-48); mirror the version-gated --disable-ngc flag here
so aws-ofi-nccl installs regardless of the base-image NGC detection.

Only affects pytorch - sglang, vllm, and vllm-omni use the common
installer via their own paths.

Both fetch_wheels.sh and upload_wheels.sh keyed the S3 wheel cache
by ${PKG}/cu${MAJOR}${MINOR}/${FNAME}, omitting the torch version the
wheel was compiled against. Since pytorch 2.12 and 2.13 both pin
flash_attn 2.8.3 + cuda 13.0.2, the 2.13 build fetched 2.12's cached
wheel and loaded it into torch 2.13 - producing an undefined symbol
at import (c10::impl::cow::materialize_cow_storage, removed in
torch 2.13).

Key by torch major.minor as well, so cross-torch-version cache reuse
is impossible by construction. transformer-engine-torch would hit the
same bug at the next torch bump.

Note: stale cu130/torch212 wheels remain in s3://dlc-cicd-wheels/
under the old key format. They're now orphaned and unreferenced -
safe to delete via a separate S3 cleanup, or leave as is.

TE 2.16.1 calls cublasLtGroupedMatrixLayoutInit_internal which requires
libcublasLt.so.13.3, but our runtime base image (nvidia/cuda:13.0.2-
runtime-amzn2023) ships an older cuBLASLt. The build succeeds against
devel headers but fails to load at runtime with:

  ImportError: /path/libtransformer_engine.so: undefined symbol
  cublasLtGroupedMatrixLayoutInit_internal, version libcublasLt.so.13

TE 2.13.0 predates the cuBLAS 13.3 dependency (grouped GEMM with
on-device group sizes landed in TE 2.14 via NVIDIA/TransformerEngine
#2748) and is torch-2.13 compatible. Bump base CUDA to 13.3.x-amzn2023
in a follow-up once that image is available on Docker Hub.

Reverts commit e1c8715. That commit rolled TE back to 2.13.0 based on
deep-analysis that assumed the cuBLAS 13.3 grouped-GEMM dependency was
introduced in TE 2.14+. That analysis was wrong: NVIDIA's PyPI
distribution transformer_engine_cu13-2.13.0 is a PRECOMPILED wheel
built against cuBLAS 13.3, so the undefined symbol persists on 2.13.0
too.

Follow-up commits pin TE to the latest release + force source build
against our cuBLAS 13.0, which excludes the grouped-GEMM code path via
the compile-time #if CUBLAS_VERSION >= 130300 guard.

Restores TE currency to the latest release. Combined with the source-
build change in the next commit, this uses NVIDIA's newest features
while avoiding the cuBLAS 13.3 binary dep that ships in their PyPI
wheels.

Compat verified: torch 2.13 (declared torch>=2.1 in build_tools/
pytorch.py), CUDA 13.0 build-time headers (source guards on runtime
symbols), cuDNN 9.x bundled with torch wheel.

nvidia/cuda:13.3.0-{base,devel,runtime}-amzn2023 are now published on
Docker Hub. Bumping the base ships cuBLAS 13.3 in the runtime, which
resolves the undefined-symbol import error on transformer_engine 2.17.0:

  OSError: libtransformer_engine.so: undefined symbol:
  cublasLtGroupedMatrixLayoutInit_internal, version libcublasLt.so.13

The symbol was added in cuBLAS 13.3. NVIDIA's precompiled
transformer_engine_cu13-2.17.0 wheel expects it, but our previous base
(nvidia/cuda:13.0.2-runtime-amzn2023) shipped cuBLAS 13.0.

This replaces the source-build workaround we were considering
(git+https install of TE with --no-build-isolation). Now we can consume
NVIDIA's precompiled wheel directly and stay current on TE.

Also updates prod_image tag from cu130 -> cu133 to match the new base.

Both CVEs are actively exploited per NVD:

- CVE-2026-7246 (click) - fixed in 8.3.3
- CVE-2026-55379 (pillow) - fixed in 12.3.0

Follows the same transitive-pin pattern as commit 7f3372f (soupsieve).

… errors

Two related fixes to the CUDA 13.3 rebuild path:

1. NCCL headers changed shape in the torch 2.13 wheel - the nccl_device/
   subdirectory (containing coop.h) was missed by the previous flat
   `cp *.h` glob. Copy the include tree recursively so TE's source
   build can find <nccl_device/coop.h>.

2. Remove the `|| true` at the end of the TE install RUN block. It was
   masking source-build compile failures - TE would silently be
   missing from the runtime image and only surface as
   `ModuleNotFoundError` in downstream tests. Fail loudly at build time.

nvidia-container-toolkit 1.19.1 (current AWS DLAMI version) has a stale
semver parser and compat table that reject `com.nvidia.cuda.requires=cuda>=13.3`
even though the runner's actual driver (580.159.04) exceeds CUDA 13.3's
declared minimum (580.65). Container prestart hook errors with
"Auto-detected mode as 'legacy'" and single-gpu tests never run.

Upstream nvidia-container-toolkit v1.20.0-rc.1 (2026-07-09) contains the
fix (Masterminds/semver parser + updated compat tables), but no AWS AMI
has consumed the new version yet.

Bypass with NVIDIA_DISABLE_REQUIRE=true until the DLAMI ships toolkit
>=1.20. The bypass only skips the label version-check; all other
container startup logic (GPU mount, driver library injection) remains
active. Revert this line once the DLAMI updates.

Commit 51a936c removed || true broadly to stop masking TE build
failures. But one `cp` in that RUN block is legitimately best-effort:
the pip wheel-cache save-off at the tail of the source-build branch.

TE compile and install were succeeding on 2.13-ec2-cuda in PR CI
run 29302027689, but the trailing
`cp /root/.cache/pip/wheels/*/*.whl /tmp/built_wheels/` returned
non-zero (glob mismatch or empty cache dir) and failed the whole
Dockerfile stage.

Restore || true only on that specific cache-save line. Actual TE
build failures still fail the build loudly — that fix from 51a936c
stays intact.

Adds an inline TODO comment near the bypass added in 30c2a47 so
future maintainers can identify when and how to remove it without
spelunking commit history.

The wheel-cache entry at s3://dlc-cicd-wheels/wheels/flash_attn/cu133/torch213/
got poisoned during earlier CI iterations on this PR — it contains a
flash-attn wheel compiled against torch 2.12 ABI, causing
`undefined symbol: c10::impl::cow::materialize_cow_storage` at runtime
in every 2.13 GPU test since (run 29302504659, jobs 86997450461 etc).

Adds a `build.skip_wheel_cache` config knob honored by pre_build.sh —
when set, forces a source build regardless of what's in S3. Enables it
on the two 2.13 GPU configs for one CI cycle. Source-built wheels then
overwrite the poisoned S3 entries via post_build.sh's normal upload
path. Follow-up commit removes the skip_wheel_cache flags after CI is
green so subsequent builds hit the freshly-populated (clean) cache.

The pre_build.sh mechanism stays as reusable infrastructure — future
cache-corruption incidents can be resolved by flipping the same flag.

…onfigs

Reverts the YAML additions from c0e3285 (skip_wheel_cache: "true" on
2.13-{ec2,sagemaker}-cuda.yml). The wheel cache at cu133/torch213/
has been refreshed with valid torch-2.13-ABI wheels via one source-
build cycle. Re-enable normal cache fetch for subsequent builds.

The pre_build.sh SKIP_WHEEL_CACHE mechanism itself stays as reusable
infrastructure — future cache-corruption incidents can be resolved
by flipping the same flag.

Reverts the pre_build.sh half of c0e3285. The mechanism was inert
(no config uses skip_wheel_cache flag any longer, that was reverted
in 0851145), but keep the script minimal — no unused infrastructure.

If wheel-cache corruption happens again, we can add the mechanism
back at that time.

Restores the SKIP_WHEEL_CACHE guard in pre_build.sh (previously reverted
in cbca116) and extends it: when a config sets skip_wheel_cache:"true",
we now also export FORCE_WHEEL_CACHE_UPLOAD=true so upload_wheels.sh
knows to overwrite any existing S3 wheel at the target key. Needed to
replace a poisoned flash-attn wheel at cu133/torch213/ that the prior
skip-only mechanism couldn't dislodge (upload was blocked by "already
cached" guard, leaving S3 untouched).

When FORCE_WHEEL_CACHE_UPLOAD=true (set by pre_build.sh when
skip_wheel_cache is enabled on a config), upload_wheels.sh overwrites
any existing S3 wheel at the target key instead of skipping. Used to
replace poisoned cache entries with freshly source-built wheels.

Forces one CI cycle to source-build flash-attn (and TE) then overwrite
the poisoned S3 cache entry at wheels/flash_attn/cu133/torch213/. The
existing wheel is torch-2.12 ABI (confirmed via undefined-symbol at
import: c10::impl::cow::materialize_cow_storage), preventing 2.13 GPU
tests from running.

Follow-up commit removes this flag after CI is green so subsequent
builds hit the freshly overwritten (clean) cache.

test_import_gpu[transformer_engine] fails on the CPU-only default-runner
that unit-test runs on because TE 2.17 dlopens libcuda.so.1 at import
time; libcuda is only mounted into containers via nvidia-container-cli
when a GPU is present.

The test is correctly gated with @pytest.mark.skipif(not IS_CUDA) — that
gate is about the IMAGE variant (CUDA vs CPU), not about the test
runner's GPU availability. Moving both parametrized cases (flash_attn,
transformer_engine) into single_gpu/ where the workflow runs on
x86-g6xl-runner with --gpus all gives them a real driver mount at
import time.

flash_attn passed on the CPU fleet only because it defers libcuda dlopen
until the first CUDA call — the import itself succeeded. Moving it too
keeps GPU-only import checks in a single suite and avoids relying on
that lazy-loading behavior.

The IS_CUDA skip is preserved as a belt-and-suspenders guard for any
future CPU-variant coverage that reuses the single_gpu suite.

Unit suite retains COMMON_PACKAGES (torch, torchvision, torchaudio,
deepspeed, boto3, requests, yaml, packaging) — none of which need a
GPU driver at import.

…onfigs

The wheel cache at wheels/flash_attn/cu133/torch213/ and
wheels/transformer_engine_torch/cu133/torch213/ has been overwritten
with clean torch-2.13-ABI wheels via the FORCE_WHEEL_CACHE_UPLOAD
mechanism (a67ea66 + b5b0a8d). PR #6365 2.13 unit-test and
single-gpu-test both pass. Re-enable normal cache fetch for
subsequent builds.

Reverts a67ea66. Cache poisoning has been resolved; keep pre_build.sh
minimal. Future incidents can restore this mechanism on-demand.

Empty commit. S3 wheel cache and ECR image tags for 2.13 GPU builds
were manually cleaned up to recover from a poisoned flash-attn wheel
(torch-2.12 ABI at a torch213-labeled S3 key). This commit forces a
fresh workflow run so build -> test dependencies resolve in order
rather than being racing rerun-failed re-queues.

Adds an inline comment inside the builder-flash-attn RUN block to
change its BuildKit stage hash, forcing a fresh source build against
the current torch 2.13 pin.

Background: earlier CI cycles on this branch cached a flash-attn wheel
that was compiled against torch 2.12 (undefined symbol
c10::impl::cow::materialize_cow_storage at import time on torch 2.13).
The bad wheel persisted through S3 delete + ECR tag delete because
BuildKit's stage cache lives across untagged image layers and the
fleet host's local buildkitd cache, both of which we cannot enumerate
and clean cleanly. Changing the RUN command text is the only way to
guarantee BuildKit re-executes the stage.

The comment is intentionally verbose and self-documenting so future
readers understand its purpose and don't reflexively delete it.

…n stage cache

The `#` comment block added in ad73db2 didn't invalidate BuildKit's
stage cache — Docker's Dockerfile parser strips `#` comment lines
from RUN commands before hashing, so the RUN command text was
unchanged and BuildKit continued serving the poisoned wheel from
cache (evidenced by 18s cached build on run 29384147865).

Replace the inline comment with a `: "string" && \` shell no-op —
a real POSIX shell command that Docker's parser MUST preserve. The
`:` builtin evaluates its args and returns success (zero runtime
cost). BuildKit sees new RUN text → cache miss → re-executes stage
against current torch 2.13 pin.

The explanatory comment is moved ABOVE the RUN (Dockerfile-level,
not shell-level), so future readers understand why the no-op line
exists.

The S3 wheel at wheels/flash_attn/cu133/torch213/ contains a
flash-attn wheel compiled against torch 2.12 (nm demangle confirms
undefined symbol c10::impl::cow::materialize_cow_storage — removed
in torch 2.13.0). Every CI cycle:
1. pre_build.sh fetches the poisoned wheel from S3 → /tmp/wheels/
2. Dockerfile's builder-flash-attn RUN finds the wheel → installs it
   (never takes source-build branch)
3. post_build.sh re-uploads the same poisoned wheel to S3

Temporarily force the source-build path unconditionally to produce
a fresh torch-2.13-ABI wheel. Add ===DIAG=== markers to log:
- torch version + path visible at flash-attn build time
- nvcc + CUDA_HOME + CUDNN_HOME
- contents of /tmp/wheels/
- undefined symbols in the newly-built wheel (proves source build
  produces correct ABI)

Revert this commit after CI is green and S3 is repopulated with a
verifiably clean wheel.

Commit a19b3b3 used a multi-line Python heredoc inside the RUN's
diagnostic block. Docker's Dockerfile parser interprets each newline
in the RUN as a potential new instruction, so `import subprocess...`
was read as a Dockerfile instruction — build failed at parse time
with "line 117: unknown instruction: import".

Replace with pure shell: unzip the built wheel + nm to check for
the poisoned symbol. No Python multi-line block.